Facial Expression Recognition Research Articles

Currency Classification Using Deep Learning

Among the uses of machine learning is the recognition of facial expressions. Based on the features that are derived from an image, it assigns a facial expression to one of the classes of facial expressions. Convolutional Neural Network (CNN) is a classification technique that may also be used to identify patterns in an image. We used the CNN approach to identify facial expressions in our proposed study. To increase the precision of facial emotion recognition, the wavelet transform is used after CNN processing. Seven distinct facial expressions are included in the facial expression image dataset that was obtained from Kaggle. The findings of the face expression recognition experiment utilizing CNN and wavelet transform show that the accuracy is improved, and the output is audible. Index Terms: Facial and Convolutional Neural Nets

Effects of ebselen addition on emotional processing and brain neurochemistry in depressed patients unresponsive to antidepressant medication

Lithium is an effective augmenting agent for depressed patients with inadequate response to standard antidepressant therapy, but numerous adverse effects limit its use. We previously reported that a lithium-mimetic agent, ebselen, promoted a positive emotional bias—an indicator of potential antidepressant activity in healthy participants. We therefore aimed to investigate the effects of short-term ebselen treatment on emotional processing and brain neurochemistry in depressed patients with inadequate response to standard antidepressants. We conducted a double-blind, placebo-controlled 7-day experimental medicine study in 51 patients with major depressive disorder who were currently taking antidepressants but had an inadequate response to treatment. Participants received either ebselen 600 mg twice daily for seven days or identical matching placebo. An emotional testing battery, magnetic resonance spectroscopy and depression and anxiety rating scales were conducted at baseline and after seven days of treatment. Ebselen did not increase the recognition of positive facial expressions in the depressed patient group. However, ebselen increased the response bias towards fear emotion in the signal detection measurement. In the anterior cingulate cortex, ebselen significantly reduced the concentrations of inositol and Glx (glutamate+glutamine). We found no significant differences in depression and anxiety rating scales between visits. Our study did not find any positive shift in emotional bias in depressed patients with an inadequate response to antidepressant medication. We confirmed the ability of ebselen to lower inositol and Glx in the anterior cingulate cortex. These latter effects are probably mediated through inhibition of inositol monophosphatase and glutaminase respectively.

A Semi‐Interpenetrating Poly(Ionic Liquid) Network‐Driven Low Hysteresis and Transparent Hydrogel as a Self‐Powered Multifunctional Sensor

AbstractConductive hydrogels are gaining significant attention as promising candidates for the fabrication materials for flexible electronics. Nevertheless, improving the tensile properties, hysteresis, durability, adhesion, and electrochemical properties of these hydrogels remains challenging. This work reports the development of a novel semi‐interpenetrating network poly(ionic liquid) hydrogel named PATV, via the in situ polymerization of acrylamide, N‐[Tris(hydroxymethyl)methyl] acrylamide, and 1‐vinyl‐3‐butylimidazolium tetrafluoroborate. The density functional theory calculations reveal that the poly(ionic liquid) in the hydrogel network acts as physical cross–linking points to construct hydrogen‐bond networks. Furthermore, the hydrogen‐bond networks dissipate energy efficiently and quickly, and thus stress concentration and hysteresis are avoided. The prepared hydrogel has a low hysteresis (9%), high tensile properties (900%), fast response (180 ms), high sensitivity (gauge factor = 10.4, pressure sensitivity = 0.14 kPa−1), and wide sensing range (tensile range: 1–600%, compression range: 0.1–20 kPa). A multifunctional sensor designed based on the designed hydrogel enables real‐time, rapid, and stable response‐ability for the detection of human movement, facial expression recognition, pronunciation, pulse, handwriting, and Morse code encryption. Furthermore, the assembled triboelectric nanogenerator displays an excellent energy harvesting capability, thus highlighting its potential application in self‐powered flexible wearable electronic devices.

The role of emotional factors in face processing abilities in autism spectrum conditions

Facial emotion recognition is considered atypical in individuals with autism spectrum conditions (ASC), but emotion recognition abilities vary widely in autistic people, and there are inconsistent findings on the causes of these differences. Research indicates alexithymia may result in facial emotion recognition differences in ASC. Alternatively, mood disorders have been linked to atypical facial emotional expression recognition abilities in neurotypical adults. Investigating both the effects of alexithymia and mood disorders (depression and anxiety) is necessary to establish which of these factors may cause atypical facial emotion recognition in ASC. This study aimed to examine whether alexithymia or mood disorder symptomology is a predictor of atypical facial emotion recognition in individuals with ASC. Ninety-eight non-autistic adults and 80 autistic adults were recruited. Participants completed an online facial processing task to examine emotion and identity recognition abilities, the AQ-28, the TAS-20, and the HADS to measure autism severity, alexithymia symptoms, and depression and anxiety symptoms. Regression-based analyses found that autistic traits and autistic group membership did not predict facial emotion processing abilities after accounting for demographic variables, alexithymia and mood disorders: however, neither alexithymia nor mood disorder symptoms predicted variance in face processing abilities either. Our results concur with previous meta-analyses of facial emotion processing in autism spectrum disorder which report that studies do not always report deficits in face processing in autism: our findings are also not supportive of the model that argues that alexithymia explains facial emotion processing difficulties in autism.

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

A joint local spatial and global temporal CNN-Transformer for dynamic facial expression recognition

Unlike conventional video action recognition, Dynamic Facial Expression Recognition (DFER) tasks exhibit minimal spatial movement of objects. Addressing this distinctive attribute, we propose an innovative CNN-Transformer model, named LSGTNet, specifically tailored for DFER tasks. Our LSGTNet comprises three stages, each composed of a spatial CNN (Spa-CNN) and a temporal transformer (T-Former) in sequential order. The Spa-CNN extracts spatial features from images, yielding smaller-sized feature maps to alleviate the computational complexity for subsequent T-Former. The T-Former integrates global temporal information from the same spatial positions across different time frames while retaining the feature map dimensions. The alternating interplay between Spa-CNN and T-Former ensures a continuous fusion of spatial and temporal information, leading our model to excel across various real-world datasets. To the best of our knowledge, this is the first method to address the DFER challenge by focusing on capturing the temporal changes in muscles within local spatial regions. Our method has achieved state-of-the-art results on multiple in-the-wild datasets and datasets under laboratory conditions.

자폐 범주성 장애 아동의 얼굴 표정 인식 관련 중재연구의 동향 및 과제

This study delves into the evolving landscape of interventions aimed at improving facial expression recognition interventions for children with autism spectrum disorder (ASD). Research was conducted from 2014 to 2023, and analyzed 23 selected academic journal articles published abroad were analyzed. The results revealed the following findings. The findings were as follows: Research on recognition interventions was consistently conducted from 2014 to 2022. Second, most study participants fell within the age bracket of 6 to 12 years old. Third, the interventions were predominantly conducted by professionals in schools and university research laboratories. Fourth, the most commonly used intervention strategy was technology-based approaches, and most of the studies presented facial expressions dynamically. In the most studies, the focus primarily rested on six basic emotions were set as dependent variables, while an additional 18 emotions were taught. The method of choosing emotions by looking at pictures was most commonly used to when measure dependent variables.

Emotion and Gesture Recognition

Human Gestures and emotions play an important role in interpersonal relationships. The automatic recognition of emotions and gestures has been an active research topic from early eras. Emotions are reflected from facial expression, speech and gesture of the body. Hence understanding of emotion and gesture has high importance in interaction between humans(human-human) as well as between human and machine communication.In this system captured images are compared with the trained dataset available in the database and then display emotional state and gesture. For this system, fer13 (Facial Expressions Recognition 2013 dataset) and VGG16 (Visual Geometry Group) datasets are used to compare with captured images. In this system emotion and gesture recognition, both will run simultaneously. EDA(Exploratory Data Analysis) is used to analyse and validate the train data for both emotion as well as gesture dataset.

Facial Expression Based Restaurant Scoring System

Abstract: Automated becoming unmanned restaurants have become more and more common in recent years. Customers' opinions cannot be directly seen in order to learn about their encounters with the idea of a restaurant because staff members are not there. This research provides a rating system based on already trained convolutional neural network (CNN) algorithms for face expression recognition. It consists of an AI server that has been trained, a web server, and an Android mobile application. Ratings are intended for both the cuisine and the setting. At the moment, the scoring system offers three expressions: neutral, disappointed, and satisfied.

Twin attention based multi-task convolutional bidirectional long short term memory for facial expression recognition

Twin attention based multi-task convolutional bidirectional long short term memory for facial expression recognition

Family Emotional Decorative Paintings: Interaction Design for Communication

This paper describes an approach applied to decorative home paintings: an emotional interaction approach. The study utilizes the theory of constructed emotion and the Wizard of Oz research methodology to extend the user's emotional experience in home scenarios with emotional self-reflection and regulation. Family Emotion Decorative Painting uses the family-shared photo library as the image base data, obtains the emotion type through facial expression and voice emotion recognition, and correlates the moment's emotion with the emotion reacted in the picture. Through qualitative research, we found that family emotional decorative painting can bring emotional arousal and soothing to participants. Family Emotional Decorative Paintings also provoked reflection on technology in and around the home. These findings suggest several opportunities, such as designing for anticipation, better supporting reflection on the past. The study also allowed the team to question the traditional interaction design view that more frequent and prolonged human-computer interactions are a measure of good design. The team will next focus on ways for users to engage in novel and meaningful ways outside of the application.

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Facial expression recognition ability and its neuropsychological mechanisms in children with attention deficit and hyperactive disorder.

Attention deficit and hyperactive disorder (ADHD) is a chronic neurodevelopmental disorder characterized by inattention, hyperactivity-impulsivity, and working memory deficits. Social dysfunction is one of the major challenges faced by children with ADHD. It has been found that children with ADHD can't perform as well as typically developing children on facial expression recognition (FER) tasks. Generally, children with ADHD have some difficulties in FER, while some studies suggest that they have no significant differences in accuracy of specific emotion recognition compared with typically developing children. The neuropsychological mechanisms underlying these difficulties are as follows. First, neuroanatomically. Compared to typically developing children, children with ADHD show smaller gray matter volume and surface area in the amygdala and medial prefrontal cortex regions, as well as reduced density and volume of axons/cells in certain frontal white matter fiber tracts. Second, neurophysiologically. Children with ADHD exhibit increased slow-wave activity in their electroencephalogram, and event-related potential studies reveal abnormalities in emotional regulation and responses to angry faces when facing facial stimuli. Third, psychologically. Psychosocial stressors may influence FER abilities in children with ADHD, and sleep deprivation in ADHD children may significantly increase their recognition threshold for negative expressions such as sadness and anger. This article reviews research progress over the past three years on FER abilities of children with ADHD, analyzing the FER deficit in children with ADHD from three dimensions: neuroanatomy, neurophysiology and psychology, aiming to provide new perspectives for further research and clinical treatment of ADHD.

Assessing children's outdoor thermal comfort with facial expression recognition: An efficient approach using machine learning

In this study, children's physiological indices and facial expressions during activities with distinct intensities in outdoor open spaces were real-timely measured. Correspondingly, meteorological measurements and questionnaire surveys were conducted to explore change laws of children's physiological feedback, facial expressions, and subjective perception. Then, a predictive model of children's outdoor thermal sensations was built with the coupling relations of physiological indices and facial expressions. Results showed that the skin temperature on the face tended to increase as thermal stress increased but gradually declined with increasing activity intensity. Mean ear skin temperature (Tear) was the most sensitive, followed by mean cheek temperature (Tcheek). When conducting moderate and vigorous intensity activities, the heart rate (HR) increased significantly within 5 min of exercise and then stabilized. The HR increase in vigorous intensity activities was greater than that during moderate intensity activities. With an increase in thermal stress and activity intensity, the proportion of negative emotions (NegEmo) in facial expressions gradually increased. Sadness and disgust expressions both increased significantly, followed by fear and anger. Thus, Tear and NegEmo expressed facially were important indices to predict children's outdoor thermal sensation. Finally, the predictive model for children's outdoor thermal sensation based on facial expressions, Tear, and HR allowed for non-invasive data collection was built integrating machine learning, and its accuracy reached as high as 97.1 %. Our results propose a faster feedback method for children's thermal perception based on facial expression recognition, which can real-timely monitor thermal comfort for children, ensuring their outdoor thermal health and safety.

Electroencephalogram-Based Facial Gesture Recognition Using Self-Organizing Map.

Brain-computer interfaces (BCIs) allow information to be transmitted directly from the human brain to a computer, enhancing the ability of human brain activity to interact with the environment. In particular, BCI-based control systems are highly desirable because they can control equipment used by people with disabilities, such as wheelchairs and prosthetic legs. BCIs make use of electroencephalograms (EEGs) to decode the human brain's status. This paper presents an EEG-based facial gesture recognition method based on a self-organizing map (SOM). The proposed facial gesture recognition uses α, β, and θ power bands of the EEG signals as the features of the gesture. The SOM-Hebb classifier is utilized to classify the feature vectors. We utilized the proposed method to develop an online facial gesture recognition system. The facial gestures were defined by combining facial movements that are easy to detect in EEG signals. The recognition accuracy of the system was examined through experiments. The recognition accuracy of the system ranged from 76.90% to 97.57% depending on the number of gestures recognized. The lowest accuracy (76.90%) occurred when recognizing seven gestures, though this is still quite accurate when compared to other EEG-based recognition systems. The implemented online recognition system was developed using MATLAB, and the system took 5.7 s to complete the recognition flow.

Synchrony or asynchrony: development of facial expression recognition from childhood to adolescence based on large-scale evidence.

The development of facial expression recognition ability in children is crucial for their emotional cognition and social interactions. In this study, 510 children aged between 6 and 15 participated in a two forced-choice task of facial expression recognition. The findings supported that recognition of the six basic facial expressions reached a relatively stable mature level around 8-9 years old. Additionally, model fitting results indicated that children showed the most significant improvement in recognizing expressions of disgust, closely followed by fear. Conversely, recognition of expressions of happiness and sadness showed slower improvement across different age groups. Regarding gender differences, girls exhibited a more pronounced advantage. Further model fitting revealed that boys showed more pronounced improvements in recognizing expressions of disgust, fear, and anger, while girls showed more pronounced improvements in recognizing expressions of surprise, sadness, and happiness. These clear findings suggested the synchronous developmental trajectory of facial expression recognition from childhood to adolescence, likely influenced by socialization processes and interactions related to brain maturation.

Lightweight facial expression estimation for mobile computing in portable device

AbstractFacial expression recognition has been studied for many years, especially with the development of deep learning. However, the existing researches still have the following two issues. Firstly, the intensity of facial expression is neglected. Secondly, the deep learning based approaches cannot be directly deployed in the devices with limited resources. In order to tackle these two issues, this paper proposes a lightweight facial expression estimation method using a shallow ordinal regression algorithm, which is deployed in a portable smart device for mobile computing in IoTs. Compared with classification based facial expression recognition methods, ordinal regression considers the intensity of facial expression to achieve better mean absolute error (MAE), which is validated by experiments on several public facial expression datasets. The simulation in portable device also demonstrates its effectiveness for mobile computing.

Visual Field Restriction in the Recognition of Basic Facial Expressions: A Combined Eye Tracking and Gaze Contingency Study

Uncertainties and discrepant results in identifying crucial areas for emotional facial expression recognition may stem from the eye tracking data analysis methods used. Many studies employ parameters of analysis that predominantly prioritize the examination of the foveal vision angle, ignoring the potential influences of simultaneous parafoveal and peripheral information. To explore the possible underlying causes of these discrepancies, we investigated the role of the visual field aperture in emotional facial expression recognition with 163 volunteers randomly assigned to three groups: no visual restriction (NVR), parafoveal and foveal vision (PFFV), and foveal vision (FV). Employing eye tracking and gaze contingency, we collected visual inspection and judgment data over 30 frontal face images, equally distributed among five emotions. Raw eye tracking data underwent Eye Movements Metrics and Visualizations (EyeMMV) processing. Accordingly, the visual inspection time, number of fixations, and fixation duration increased with the visual field restriction. Nevertheless, the accuracy showed significant differences among the NVR/FV and PFFV/FV groups, despite there being no difference in NVR/PFFV. The findings underscore the impact of specific visual field areas on facial expression recognition, highlighting the importance of parafoveal vision. The results suggest that eye tracking data analysis methods should incorporate projection angles extending to at least the parafoveal level.

Modelling appearance variations in expressive and neutral face image for automatic facial expression recognition

AbstractIn automatic facial expression recognition (AFER) systems, modelling the spatio‐temporal feature information in a specific manner, coalescing, and its effective utilization is challenging. The state‐of‐the‐art studies have examined integrating multiple features to enhance the recognition rate of AFER systems. However, the feature variations between expressive and neutral face images are not fully explored to identify the expression class. The proposed research presents an innovative approach to AFER by modelling appearance variations in both expressive and neutral face images. The prominent contributions of the work are developing a novel and hybrid feature space by integrating the discriminative feature distribution derived from expressive and neutral face images; preserving the highly discriminative latent feature distribution using autoencoders. Local binary pattern (LBP) and histogram of oriented gradients (HOG) are the feature descriptors employed to derive the discriminative texture and shape information, respectively. The component‐based approach is employed, wherein the features are derived from the salient facial regions instead of the whole face. The three‐stage stacked deep convolutional autoencoder (SDCA) and multi‐class support vector machine (MSVM) are employed to address dimensionality reduction and classification, respectively. The efficacy of the proposed model is substantiated by empirical findings, which establish its superiority in terms of accuracy in AFER tasks on widely recognized benchmark datasets.

Quantum-Inspired Vortex Search Algorithm with Deep Neural Networks for Multi-Pose Facial Expression Recognition

Quantum-Inspired Vortex Search Algorithm with Deep Neural Networks for Multi-Pose Facial Expression Recognition